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Turning  Virtual  Reality  into  a  More  Realistic  and 
Effective  Infantry  Training  Ground 
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With  the  invention  of  Pointman,  a  new  desktop  control, 
trainees  can  now  realistically  perform  actions  in 
VBS2,  the  training  simulator  used  by  the  U.S.  Marine 
Corps.  NRL’s  Immersive  Simulation  Section  has  developed 
a  novel  user  interface  that  will  provide  the  Corps  with  the 
capability  to  supplement  live  training  by  means  of  a  portable 
computer-based  system  that  can  train  dangerous  situations 
without  risking  life  or  limb. 

For  a  generation  who  grew  up  with  game  controllers 
practically  as  appendages,  how  does  a  new,  more  realistic 
interface  better  accomplish  the  difficult  task  of  bringing 
the  battlefield  to  the  desktop?  By  making  the  simulated 
movement  more  natural,  fluid,  and  reflective  of  the  real-world 
engagement  experience;  by  incorporating  correct,  realistic 
tactical  movements  into  the  virtual  reality  experience;  and  by 
using  enhanced  inputs  (dual  joysticks;  sliding  foot-pedals 
to  simulate  stepping,  squatting,  and  lying  flat;  and  a  head¬ 
tracking  sensor). 

Each  of  these  features  is  novel  and  increases  the 
effectiveness  of  the  training.  For  instance,  applying  dual 
joysticks  in  a  novel  way  allows  the  user  to  control  the  heading 
(the  direction  that  the  body  of  the  avatar,  the  virtual  soldier, 
faces)  and  course  (the  direction  of  movement  of  the  avatar) 
independently.  That  means  that  the  avatar  can  turn  to  look  or 
shoot  without  redirecting  its  movement  along  a  path,  enabling 
both  scanning  for  enemies  and  coordinating  actions  with 
squad  members. 

In  combat,  the  pointman  leads  the  patrol  and  is  the  first  to 
encounter  the  enemy.  With  NRL’s  Pointman,  today’s  soldiers 
can  get  additional  assistance  in  practicing  the  tactical  team 
movement  involved  with  dismounted  combat  without  risking 
injury  and  without  DoD  incurring  extra  logistical  expense.  This 
invention,  then,  even  more  importantly  than  saving  money, 
can  help  save  our  soldiers’  lives.  And  isn’t  that  the  point,  after 
all? 
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For  more  than  a  decade,  the  Immersive  Simulation  Section  has  developed  novel  user  interfaces  for  U.S.  Marine  Corps 
(USMC)  infantry  training  simulators  that  let  users  interact  in  the  3D  virtual  world  with  close  to  the  same  abilities 
and  constraints  people  have  in  the  real  world,  in  terms  of  moving  through  the  environment  and  coordinating  with 
teammates.  Pointman  is  a  new  desktop  control  interface  that  combines  a  dual- joystick  gamepad,  head  tracking,  and  sliding 
foot-pedals  (inexpensive  rudder  pedals  used  with  flight  simulator  games)  that,  for  the  first  time,  gives  users  the  capability 
to  apply  correct  tactical  infantry  movements  using  a  desktop  system.  The  USMC  has  adopted  VBS2  (Virtual  BattleSpace-2), 
an  infantry  and  combined  arms  desktop  training  simulator  to  supplement  live  training.  VBS2  is  a  product  of  Bohemia 
Interactive  and  will  be  used  within  the  framework  of  the  USMC’s  Deployable  Virtual  Training  Environment  (DVTE). 
Training  simulators  present  dangerous  situations  without  risking  injury  or  accidents.  They  can  represent  any  location 
that  has  been  modeled,  have  a  small  footprint,  and  are  deployable.  They  are  less  costly  because  they  do  not  require  travel 
to  physical  training  sites  and  do  not  use  consumables.  The  USMC  recognizes  the  advantages  of  Pointman  for  increasing 
training  effectiveness  and  has  endorsed  Pointman’s  integration  with  VBS2  through  the  Office  of  Naval  Research’s  Rapid 
Technology  Transition  (RTT)  Program  and  in  cooperation  with  the  USMC  Program  Manager  Training  Systems  office 
(PMTRASYS)  in  Orlando,  Florida. 


INTRODUCTION 

Desktop  training  simulators  have  the  potential  to 
provide  training  anytime  and  anywhere.  A  multiplayer 
training  simulator  can  engage  a  whole  squad  in  one 
action.  A  missing  component  in  today’s  desktop  simu¬ 
lators  is  a  user  interface  controller  that  allows  trainees 
to  execute  realistic  military  tactics,  techniques,  and 
procedures.  Tactical  movement  relies  heavily  on  the 
ability  to  look  around  (scan)  while  moving  along  any 
chosen  path.  Figure  1  illustrates  the  characteristics  of 
tactical  movement  for  moving  around  a  corner  in  a 
high- threat  environment.  The  objective  is  to  clear  the 
corner  while  minimizing  the  person’s  exposure  to  a 
threat.  The  action  requires  incrementally  “pie-ing”  the 
corner,  turning  the  upper  body  and  rifle  together  as  a 
unit  to  face  the  corner,  while  moving  down  the  hallway 
and  focusing  attention  on  the  area  just  past  the  corner’s 
edge. 

The  ideal  user  interface  would  enable  the  user  to 
fully  control  the  movement  of  the  user’s  avatar  (virtual 
body)  in  the  virtual  world.  This  movement  is  character¬ 
ized  by  the  avatar’s  heading  (the  direction  in  which  the 
avatar’s  body  faces)  and  course  (the  direction  in  which 
the  avatar  moves).  Current  dual -joystick  gamepad  con- 


FIGURE  1 

Turning  the  heading  toward  a  corner  while  maintaining 
a  straight  course  is  a  tactically  correct  way  to  clear  a 
corner  (referred  to  as  pie-ing  a  corner).  Direction  of 
movement  is  represented  by  the  arrow  and  heading 
coincides  with  the  direction  of  aim. 
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trailers  for  first-person  shooting  games  make  it  difficult 
to  execute  tactical  movements  because  of  limitations  in 
the  control  mapping,  which  conflate  the  control  over 
the  avatar’s  heading  and  course.  The  problem  is  that 
the  joystick  used  to  turn  the  heading  also  redirects  the 
course.  The  right  joystick  is  a  rate  control  that  turns  the 
heading  and  course  together,  while  the  left  is  a  direc¬ 
tional  control  that  changes  the  course  relative  to  the 
heading.  Tactics  for  first-person  shooter  games  reflect 
this  control  deficiency  by  emphasizing  artificial  straf¬ 
ing  motions:  moving  sideways  and  spiraling  toward  or 
away  from  the  target. 

From  its  inception,  the  goal  in  developing  Point- 
man  was  to  create  an  interface  that  allows  the  user  to 
move  his  or  her  avatar  in  a  tactically  correct  manner. 

We  learned  a  great  deal  about  control  over  natural  and 
tactical  movement  in  developing  Gaiter,  a  full-body 
immersive  interface  that  uses  a  head-mounted  display 
to  surround  the  user  in  the  virtual  world  and  tracks 
the  user’s  body  segments  in  six  degrees  of  freedom 
to  provide  direct  interaction.1  With  Gaiter,  the  user 
steps  in  place  to  move  the  avatar  through  the  virtual 
world  and  physically  turns  to  rotate.  The  ability  to  turn 
naturally  is  central  to  Gaiter  because  it  allows  the  user 
to  turn  immediately  toward  or  away  from  any  sight  or 
sound. 

The  design  goal  for  both  Gaiter  and  Pointman  is 
for  tasks  in  the  virtual  world  to  be  performed  at  the 
same  rate,  accuracy,  and  effort  as  they  are  in  the  real 
world  to  maintain  timing  and  scale.2  Having  close  to 
the  same  capability  in  the  virtual  world  as  a  person  has 
in  the  real  world  makes  it  more  likely  that  skills  devel¬ 
oped  in  one  will  transfer  to  the  other.  For  example, 
the  cadence  set  when  stepping  in  place  with  Gaiter  or 
using  Pointman’s  sliding  foot-pedals  is  mapped  to  a 
real-world  walking  or  running  cadence  on  a  moment- 
by-moment  basis  to  give  the  user  a  realistic  sense  of 
distance  traveled. 

OVERVIEW  OF  POINTMAN 

The  Pointman  user  interface  consists  of  a  conven¬ 
tional  dual- joystick  gamepad  for  directing  motion  and 


weapons  handling;  sliding  foot-pedals  to  control  step¬ 
ping;  head  tracking  for  directing  the  view  and  aim;  and 
a  desktop  display  (Fig.  2).  The  components  combine  to 
allow  the  user  to  control  the  actions  of  the  user’s  avatar 
in  a  natural  manner. 

Unlike  conventional  gamepad  controls,  both  joy¬ 
sticks  on  the  gamepad  are  applied  as  directional  con¬ 
trols.  The  user  simply  points  the  joystick  in  the  direc¬ 
tion  the  avatar  should  move  or  turn.  Figure  3  shows  the 
mapping  of  Pointman’s  dual- joystick  gamepad.  To  turn 
smoothly,  the  right  joystick  is  pushed  forward  and  slid 
against  the  circular  outer  rim  to  continuously  redirect 
the  heading.  To  turn  quickly  in  a  known  direction,  the 
joystick  is  pushed  in  the  desired  direction  from  the 
neutral  centered  position.  The  user’s  avatar  then  turns 
at  the  maximum  rate  a  person  can  physically  turn  in 
that  situation.  The  left  joystick  controls  the  stepping 
direction  (forward,  backwards,  sideways,  or  in  any 
diagonal  direction).  The  length  of  the  avatar’s  stride 
and  stepping  cadence  are  expressed  via  sliding  foot- 
pedals,  which  mimic  a  person’s  reciprocal  foot  motion 
when  walking  or  running.  Pointman  lets  users  directly 
sense  how  far  their  avatar  has  turned  or  traveled,  even 
with  their  eyes  closed:  users  feel  their  avatars’  alignment 
through  the  position  of  the  joysticks  and  perceive  how 
far  they  have  traveled  by  the  amount  they  have  stepped. 

Using  the  right  joystick  alone  while  operating  the 
sliding  foot- pedals  advances  the  avatar  in  a  forward 
direction  relative  to  the  current  heading.  People  spend 
most  of  their  time  walking  along  paths  in  this  way  (for 
example,  down  a  hall)  and  the  design  of  Pointman 
makes  such  common  acts  easy  to  do.  Using  the  left 
joystick  alone  while  operating  the  foot-pedals  translates 
the  avatar  along  the  indicated  course  while  keeping  the 
avatar’s  heading  constant. 

The  real  power  of  Pointman  comes  when  the 
joysticks  are  jointly  engaged.  Working  together,  they 
provide  independent  control  over  heading  and  course: 
the  right  joystick  turns  the  avatar  (for  example,  to  face 
a  target)  while  the  left  sets  the  avatar’s  direction  of 
movement.  This  independent  control  over  heading  and 
course  is  what  lets  the  user  scan  while  moving  along  a 
path. 


FIGURE  2 

Pointman  user  interface  with  dual  joystick  gamepad, 
head  tracking,  and  sliding  foot-pedals. 


2009  NRL  REVIEW 


61 


FEATURED  RESEARCH 


Engaging  Only  the 
Turning  Joystick 


FIGURE  3 

Mapping  of  Pointman's  dual  joystick 
gamepad  to  control  the  avatar's 
movement. 


Weapons  handling  is  accomplished  through  button 
presses  on  the  gamepad.  One  button  is  the  trigger  and 
another  cycles  through  the  different  ways  of  carry¬ 
ing  the  rifle:  slung,  carry,  ready,  and  aim.  A  change  in 
posture  is  also  indicated  through  a  button  press,  cycling 
through  standing,  prone,  seated.  Tilting  the  foot-pedals 
(similar  to  an  accelerator  pedal)  adjusts  the  height  of  a 
posture;  for  example,  with  standing,  pushing  the  pedals 
down  moves  the  avatar  from  an  upright  posture  to  a 
crouch. 

A  three-degree-of-freedom  (orientation  only) 
tracker  or  a  six-degree-of-freedom  (orientation  and 
translation)  tracker  worn  on  the  user’s  head  provides 
control  over  viewing  and  aiming.  Head  tracking  con¬ 
trols  the  yaw  (turning  about  the  vertical  axis),  pitch 
(tilting  up  and  down),  and  roll  (tilting  left  and  right) 
of  the  avatar’s  head  and  aim  relative  to  the  joystick- 
directed  heading.  The  yaw  derived  from  the  head 
tracking  data  is  added  to  the  current  heading  to  turn 
the  view  an  additional  amount  (limited  by  how  far  the 
user  can  turn  his  or  her  head  while  seated  in  front  of 
the  desktop  display).  Since  the  degree  to  which  the  user 
can  pitch  his  or  her  head  up  and  down  while  viewing 
the  desktop  display  is  limited,  the  pitch  derived  from 
the  head  tracking  data  is  amplified  to  allow  the  avatar 
to  look  directly  up  or  down  in  the  virtual  world.  If  the 
tracker  provides  six  degrees  of  freedom,  the  user  can 
direct  the  avatar’s  upper  body  to  lean  forward,  back¬ 
ward,  and  side-to-side  independently  of  how  the  head 
is  oriented,  further  increasing  the  realism.  The  direc¬ 
tion  of  aim  is,  therefore,  linked  to  both  the  heading  set 
by  the  joystick  and  the  direction  of  view,  allowing  the 
head,  upper  body,  and  rifle  to  be  rotated  as  a  unit  to 
maintain  an  indexed  shooting  posture  as  is  done  in  the 
real  world.  With  six-degree-of-freedom  tracking,  users 
can  also  lean  to  shoot  from  behind  cover. 

USER’S  AVATAR 

The  Immersive  Simulation  Section  has  pioneered 
the  use  of  a  fully  articulated  user  avatar  continuously 
driven  in  real  time  either  by  the  user’s  own  body 


motion  or  through  control  actions.  An  articulated 
avatar  is  used  with  both  Gaiter  and  Pointman.  A  central 
part  of  the  integration  effort  is  to  add  an  articulated 
avatar  to  VBS2. 

Traditionally,  avatars  in  first-person  shooter  games, 
including  VBS2,  use  prerecorded  animation  sequences 
to  portray  the  user’s  actions.  A  user  indicates  that  the 
avatar  should  move  forward  by  deflecting  a  joystick 
or  pushing  a  button,  and  an  animation  sequence  of  a 
walking  or  running  avatar  is  played.  In  contrast,  Point- 
man  lets  the  user  directly  control  his  or  her  avatar. 

The  sliding  foot-pedals  control  the  avatar’s  stepping 
motion,  course  and  heading  are  controlled  with  the 
dual  joysticks,  and  because  the  user’s  head  is  tracked, 
the  user’s  avatar  turns  to  look  in  the  same  direction 
that  the  user’s  head  turns.  In  other  words,  with  Point- 
man,  the  avatar  moves  in  correspondence  with  the 
user’s  actions  to  an  unprecedented  degree  for  a  desktop 
control. 

Users  operate  in  first  person  and  see  the  virtual 
world  out  of  their  avatars’  eyes.  The  avatar  is  fully  ren¬ 
dered  so  the  user  sees  his  or  her  avatar’s  arms,  legs,  and 
feet.  Users  also  see  their  teammates  as  fully  articulated 
avatars  acting  in  the  virtual  world.  The  challenge  is  to 
realistically  portray  the  actions  performed  by  the  user’s 
avatar  under  direct,  continuous  control.  Being  able 
to  act  in  a  realistic  manner  in  the  virtual  world  gives 
the  user  a  sense  of  perspective,  relative  distance,  and 
presence  in  the  environment.  Watching  the  actions  of 
teammates’  avatars  gives  a  better  sense  of  their  intent. 
Having  a  finer  level  of  control  in  the  virtual  world 
leads  to  a  fuller  range  of  expression,  greater  situational 
awareness,  better  communication  among  immersed 
teammates,  and  support  for  cooperative  action. 

INTEGRATION  WITH  VIRTUAL  BATTLESPACE-2 

VBS2  currently  uses  a  keyboard  and  mouse 
interface  or  conventional  gamepad  with  thumbsticks 
and  buttons,  and  prerecorded  animations  for  avatar 
movement.  Enabling  users  to  execute  correct  tactical 
methods  in  the  virtual  world  requires  the  integration 
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of  both  the  Pointman  user  interface  and  an  articu¬ 
lated  avatar  continuously  directed  in  real  time.  NRL’s 
Immersive  Simulation  Section  has  developed  a  detailed 
VBS2- Pointman  Interface  Specification  that  allows 
Pointman  to  direct  the  user’s  avatar  in  VBS2.  The  goal 
is  to  create  a  Pointman- enhanced  version  of  the  VBS2 
training  simulator,  adding  more  realistic  interaction 
while  minimizing  latency  and  communication  over¬ 
load.  Figure  4  shows  the  user’s  avatar  as  controlled  by 
Pointman  and  rendered  in  VBS2. 


FIGURE  4 

Integration  of  Pointman  with  Virtual  BattleSpace-2.  The  left 
panel  is  the  avatar  as  rendered  in  VBS2. 


The  VBS2  integration  coincides  with  new  research 
to  extend  the  range  of  behaviors  supported  by  Point- 
man  to  meet  the  demands  of  military  action  in  urban 
terrain.  The  user’s  avatar  needs  to  not  only  stand  but 
also  crouch  and  sit  while  handling  a  rifle.  We  have 
conducted  research  in  fluid  control  over  the  avatar’s 
body  posture  and  engagement  stance  so  that  the  avatar 
can  transition  between  poses  with  the  same  timing  and 
constraint  people  use  in  the  real  world.  Figure  5  shows 
the  design  for  accomplishing  a  smooth  transition 


iel  shows  the  user's  avatar  as  controlled  by  Pointman;  the  right 


smooth  transition  between  a  high  and 
upright  and  prone  postures. 
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between  a  high  and  low  stance  in  upright  and  prone 
postures.  The  challenge  has  been  to  make  the  different 
means  of  controlling  the  avatar’s  stance  and  movement 
work  together  to  express  the  user’s  intent  in  a  natural 
manner. 

CONCLUSIONS 

Pointman  is  a  new  user  interface  that  gives  users 
the  ability  to  execute  realistic  tactical  infantry  move¬ 
ments  in  a  desktop  environment.  The  military  has  a 
growing  interest  in  using  desktop  simulators  for  train¬ 
ing  because  they  are  portable,  relatively  inexpensive, 
and  can  support  a  large  number  of  users.  Four  U.S. 
Marine  Corps  Generals  who  visited  our  lab  recognized 
Pointman’s  advantages.  While  Pointman’s  independent 
control  over  moving  and  looking  remains  the  core  of 
the  user  interface,  a  key  effort  has  been  researching 
how  to  make  the  transition  between  avatar  postures 
and  stances  more  fluid  and  natural.  The  final  result  will 
be  a  desktop  training  simulator  that  provides  realistic 
control  over  action  to  support  teaching  the  full  range 


of  tactics,  techniques,  and  procedures  used  in  infantry 
operations. 

[Sponsored  by  the  NRL  Base  Program] 
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